Papers with bias evaluation methods
Gender Bias in Masked Language Models for Multiple Languages (2022.naacl-main)
Copied to clipboard
| Challenge: | Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages. |
| Approach: | They propose to use English attribute word lists to evaluate bias in eight languages without manually annotating data. |
| Outcome: | The proposed model significantly correlates with the existing English datasets for gender bias. |
TWBias: A Benchmark for Assessing Social Bias in Traditional Chinese Large Language Models through a Taiwan Cultural Lens (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown remarkable capabilities in natural language processing, but concerns about social bias amplification remain. |
| Approach: | They propose a social bias evaluation benchmark for Traditional Chinese LLMs that integrates chat templates and diverse prompts for comprehensive bias assessment. |
| Outcome: | The proposed model incorporates chat templates and diverse prompts for comprehensive bias assessment focusing on Taiwan's cultural context and prioritizing gender and ethnicity bias evaluation. |
Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate gender bias in PLMs focus on one label out of three labels, such as neutral. |
| Approach: | They propose a bias evaluation method for PLMs that considers all the three labels of NLI task and then defines a measure based on the corresponding label output. |
| Outcome: | The proposed method can distinguish biased, incorrect inferences from non-biased incorrect infertility better than baseline, resulting in a more accurate bias evaluation. |
Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for assessing social bias in large language models (LLMs) do not capture nuanced and context-dependent nature of natural language generation. |
| Approach: | They propose a Bias Benchmark for Generation (BBG) that evaluates social bias in long-form generation by having LLMs generate continuations of story prompts. |
| Outcome: | The proposed benchmark is based on the English BBQ and Korean BBQ datasets and compares it with multiplechoice BBQ evaluation. |
Unmasking Style Sensitivity: A Causal Analysis of Bias Evaluation Instability in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to assess social biases in natural language processing models show unexpected instability when input texts undergo minor stylistic changes. |
| Approach: | They conduct a comprehensive analysis of how style transformations impact bias evaluation results . they find formal style transformation significantly affects bias scores . larger models show greater sensitivity to stylistic variations, they find . |
| Outcome: | The proposed method fails to detect appearance bias, sexual orientation bias, religious bias and religious bias in large language models. |